Columbia Kermit/kermit.zip/newsgroups/misc.20000824-20010305/[e-mail] _Sat Mar 3 11:19:58 2001.msg

Internet Message Format | 2020-01-01 | 5KB

Return-Path: <[e-mail]>
Received: from newsmaster.cc.columbia.edu (newsmaster.cc.columbia.edu [128.59.59.30])
    by fozimane.cc.columbia.edu (8.9.3/8.9.3) with ESMTP id LAA24331
    for <[e-mail]>; Sat, 3 Mar 2001 11:19:58 -0500 (EST)
Received: (from news@localhost)
    by newsmaster.cc.columbia.edu (8.9.3/8.9.3) id LAA23091
    for [e-mail]; Sat, 3 Mar 2001 11:17:53 -0500 (EST)
X-Authentication-Warning: newsmaster.cc.columbia.edu: news set sender to <news> using -f
From: [e-mail] (Frank da Cruz)
Subject: Re: FTP - loops and parameters for multifile reading
Date: 3 Mar 2001 16:17:53 GMT
Organization: Columbia University
Message-ID: <97r5fh$mhh$[e-mail]>
To: [e-mail]

In article <[e-mail]>,
Paul Nestler  <[e-mail]> wrote:
: Does anyone know how to put ftp inside a loop?  Say, a loop that
: generates a list of filenames to download?
:
: Here's the background:
: I have a problem connection I am dealing with.  When transferring a
: number of files using mget in ftp, the connection hangs and aborts.  The
: point at which the ftp session hangs varies from one session to the
: next, but so far has always allowed me to transfer a few files before it
: hangs.
:
: I looked at the man page for ftp.  The man page discusses a looping
: feature and the ability to pass a parameter (from the calling script I
: suppose).  Unfortunately, the man page does not show an example nor does
: it describe the usage in any useable detail.
:
: The man page gave me the notion I should be able to put the ftp inside a
: loop and pass the name of a file to get from the remote host.  Perhaps
: there is someway of putting the ftp inside an awk script.
:
Let's see how we would do this with the new scriptable C-Kermit FTP
client.  Let's assume an anonymous login, though it need not be, and that
the files are to be transferred in binary mode, and that the filenames
are file1 through file5:

    while true {
    ftp open somehost.com /anonymous
    if fail {
        Can't reach host           ;  Can't make connection
        sleep 10*60                ;  Wait 10 minutes and try again
        continue
    }
    ftp cd somedirectory
    if fail {
        exit 1 Fatal - can't CD to somedirectory on host
    }
        while true {
            ftp mget /binary /recover file1 file2 file3 file4 file5
            if success goto done        ;  All files downloaded: done.
            if not \v(ftp_connected) {  ;  Failed - why?
                ftp close               ;  Connection closed
                break                   ;  Go back and make a new one
            }
            ; Connection still open - try MGET again.
            echo "Download failed - trying again in 20 seconds..."
            sleep 20
        continue
        }
    }
    :done
    ftp bye                             ;  Disconnect from server
    exit 0                              ;  Exit successfully

This is a very straightforward example, with hardwired host, directory,
and filenames.  Of course these could also be variables that could be
resolved in various ways, e.g. from command-line options, with interactive
prompting, or from a file.

The loop connects to the host, cd's to the desired directory, and requests
the files in binary mode.  Each step is checked for failure and is retried
until it succeeds.  Of course you could use a counted loop rather than an
infinite loop if you wished, and you could use different sleep intervals
or none at all.

In this example, failure to connect results in a 10-minute pause (assuming
the host is down or unavailable) and then a loop restart.  You could make
it do anything else that might be more appropriate.

The magic command is MGET /BINARY /RECOVER <file-list>.  For each file in
the list:

 . If the file does not exist on the client, it is downloaded; otherwise:

 . If a file of the same name exists on the client, the client requests
   the host file's size and modification date-time; if the date-time and
   size are the same, the file is skipped; otherwise:

 . If the date-time is later, the file is downloaded, overwriting the
   client's copy (which is backed up for safety); otherwise:

 . If the size is greater, the excess part of the host file is downloaded
   and appended to the client file.

The MGET command is in an inner loop, which retries the MGET command as
long as it fails and the connection is open.  If MGET fails because the
connection was closed, the outer loop is continued and a new connection is
made.  Thus no file is transferred more than once, and if a transfer fails
in the middle of a file, it is resumed from the point of failure next time
through the loop.

For more information about the C-Kermit FTP client, see:

  http://www.columbia.edu/kermit/ftpclient.html

And for a tutorial on scripting the new FTP client, see:

  http://www.columbia.edu/kermit/ftpscript.html

The new FTP client is still in prerelease testing but should be perfectly
usable, and should be formally released soon.   A prebuilt binary for AIX
4.3.2 can be downloaded from C-Kermit 7.1 site:

  http://www.columbia.edu/kermit/ck71.html

as well as the source code and binaries for many other platforms.

- Frank